SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Mixture model</title>

</head>

Mixture model

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Not to be confused with <a href="Mixed_model" title="Mixed model">mixed model</a>.</div>

<div role="note" class="hatnote navigation-not-searchable">See also: <a href="Mixture_distribution" title="Mixture distribution">Mixture distribution</a></div>

In <a href="Statistics" title="Statistics">statistics</a>, a mixture model is a <a href="Probabilistic_model" class="mw-redirect" title="Probabilistic model">probabilistic model</a> for representing the presence of <a href="Subpopulation" class="mw-redirect" title="Subpopulation">subpopulations</a> within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observation belongs. Formally a mixture model corresponds to the <a href="Mixture_distribution" title="Mixture distribution">mixture distribution</a> that represents the <a href="Probability_distribution" title="Probability distribution">probability distribution</a> of observations in the overall population. However, while problems associated with "mixture distributions" relate to deriving the properties of the overall population from those of the sub-populations, "mixture models" are used to make <a href="Statistical_inference" title="Statistical inference">statistical inferences</a> about the properties of the sub-populations given only observations on the pooled population, without sub-population identity information. Mixture models are used for clustering, under the name <a href="Model-based_clustering" title="Model-based clustering">model-based clustering</a>, and also for <a href="Density_estimation" title="Density estimation">density estimation</a>.

Mixture models should not be confused with models for <a href="Compositional_data" title="Compositional data">compositional data</a>, i.e., data whose components are constrained to sum to a constant value (1, 100%, etc.). However, compositional models can be thought of as mixture models, where members of the population are sampled at random. Conversely, mixture models can be thought of as compositional models, where the <a href="Measure_(mathematics)" title="Measure (mathematics)">total size</a> reading population has been normalized to 1.

<div class="mw-heading mw-heading2"><h2 id="Structure">Structure</h2></div>

<div class="mw-heading mw-heading3"><h3 id="General_mixture_model">General mixture model</h3></div>

A typical finite-dimensional mixture model is a <a href="Hierarchical_Bayes_model" class="mw-redirect" title="Hierarchical Bayes model">hierarchical model</a> consisting of the following components:

<ul><li>N random variables that are observed, each distributed according to a mixture of K components, with the components belonging to the same <a href="Parametric_family" title="Parametric family">parametric family</a> of distributions (e.g., all <a href="Normal_distribution" title="Normal distribution">normal</a>, all <a href="Zipf's_law" title="Zipf's law">Zipfian</a>, etc.) but with different parameters. However, it is also possible to have a finite mixture model where each component belongs to a different <a href="Parametric_family" title="Parametric family">parametric family</a> of distributions,<a href="#cite_note-1">[1]</a> for example, a mixture of a <a href="Multivariate_normal_distribution" title="Multivariate normal distribution">multivariate normal distribution</a> and a <a href="Generalized_hyperbolic_distribution" class="mw-redirect" title="Generalized hyperbolic distribution">generalized hyperbolic distribution</a>.</li></ul>

<ul><li>N random <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a> specifying the identity of the mixture component of each observation, each distributed according to a K-dimensional <a href="Categorical_distribution" title="Categorical distribution">categorical distribution</a></li>

<li>A set of K mixture weights, which are probabilities that sum to 1.</li>

<li>A set of K parameters, each specifying the parameter of the corresponding mixture component. In many cases, each "parameter" is actually a set of parameters. For example, if the mixture components are <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distributions</a>, there will be a <a href="Mean" title="Mean">mean</a> and <a href="Variance" title="Variance">variance</a> for each component. If the mixture components are <a href="Categorical_distribution" title="Categorical distribution">categorical distributions</a> (e.g., when each observation is a token from a finite alphabet of size V), there will be a vector of V probabilities summing to 1.</li></ul>

In addition, in a <a href="Bayesian_inference" title="Bayesian inference">Bayesian setting</a>, the mixture weights and parameters will themselves be random variables, and <a href="Prior_distribution" class="mw-redirect" title="Prior distribution">prior distributions</a> will be placed over the variables. In such a case, the weights are typically viewed as a K-dimensional random vector drawn from a <a href="Dirichlet_distribution" title="Dirichlet distribution">Dirichlet distribution</a> (the <a href="Conjugate_prior" title="Conjugate prior">conjugate prior</a> of the categorical distribution), and the parameters will be distributed according to their respective conjugate priors.

Mathematically, a basic parametric mixture model can be described as follows:

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>number of mixture components</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>number of observations</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>parameter of distribution of observation associated with component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>mixture weight, i.e., prior probability of a particular component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>-dimensional vector composed of all the individual </mtext>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>component of observation </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>observation </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>probability distribution of an observation, parametrized on </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}K&=&{\text{number of mixture components}}\\N&=&{\text{number of observations}}\\\theta _{i=1\dots K}&=&{\text{parameter of distribution of observation associated with component }}i\\\phi _{i=1\dots K}&=&{\text{mixture weight, i.e., prior probability of a particular component }}i\\{\boldsymbol {\phi }}&=&K{\text{-dimensional vector composed of all the individual }}\phi _{1\dots K}{\text{; must sum to 1}}\\z_{i=1\dots N}&=&{\text{component of observation }}i\\x_{i=1\dots N}&=&{\text{observation }}i\\F(x|\theta )&=&{\text{probability distribution of an observation, parametrized on }}\theta \\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}|z_{i=1\dots N}&\sim &F(\theta _{z_{i}})\end{array}}}</annotation>

</semantics>

</math><img src="./7ca7990c1a320c9fb6295fc6fde881f81890a64c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -15.671ex; width:96.789ex; height:32.509ex;" alt="{\displaystyle {\begin{array}{lcl}K&=&{\text{number of mixture components}}\\N&=&{\text{number of observations}}\\\theta _{i=1\dots K}&=&{\text{parameter of distribution of observation associated with component }}i\\\phi _{i=1\dots K}&=&{\text{mixture weight, i.e., prior probability of a particular component }}i\\{\boldsymbol {\phi }}&=&K{\text{-dimensional vector composed of all the individual }}\phi _{1\dots K}{\text{; must sum to 1}}\\z_{i=1\dots N}&=&{\text{component of observation }}i\\x_{i=1\dots N}&=&{\text{observation }}i\\F(x|\theta )&=&{\text{probability distribution of an observation, parametrized on }}\theta \\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}|z_{i=1\dots N}&\sim &F(\theta _{z_{i}})\end{array}}}" loading="lazy"></dd></dl>

In a Bayesian setting, all parameters are associated with random variables, as follows:

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>shared hyperparameter for component parameters</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>shared hyperparameter for mixture weights</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>prior probability distribution of component parameters, parametrized on </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\theta _{i=1\dots K},\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N},F(x|\theta )&=&{\text{as above}}\\\alpha &=&{\text{shared hyperparameter for component parameters}}\\\beta &=&{\text{shared hyperparameter for mixture weights}}\\H(\theta |\alpha )&=&{\text{prior probability distribution of component parameters, parametrized on }}\alpha \\\theta _{i=1\dots K}&\sim &H(\theta |\alpha )\\{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\z_{i=1\dots N}|{\boldsymbol {\phi }}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}|z_{i=1\dots N},\theta _{i=1\dots K}&\sim &F(\theta _{z_{i}})\end{array}}}</annotation>

</semantics>

</math><img src="./7bdc75e1d0a124bc684307724e2635f53b88be8d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -15.838ex; width:106.062ex; height:32.843ex;" alt="{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\theta _{i=1\dots K},\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N},F(x|\theta )&=&{\text{as above}}\\\alpha &=&{\text{shared hyperparameter for component parameters}}\\\beta &=&{\text{shared hyperparameter for mixture weights}}\\H(\theta |\alpha )&=&{\text{prior probability distribution of component parameters, parametrized on }}\alpha \\\theta _{i=1\dots K}&\sim &H(\theta |\alpha )\\{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\z_{i=1\dots N}|{\boldsymbol {\phi }}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}|z_{i=1\dots N},\theta _{i=1\dots K}&\sim &F(\theta _{z_{i}})\end{array}}}" loading="lazy"></dd></dl>

This characterization uses F and H to describe arbitrary distributions over observations and parameters, respectively. Typically H will be the <a href="Conjugate_prior" title="Conjugate prior">conjugate prior</a> of F. The two most common choices of F are <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian</a> aka "<a href="Normal_distribution" title="Normal distribution">normal</a>" (for real-valued observations) and <a href="Categorical_distribution" title="Categorical distribution">categorical</a> (for discrete observations). Other common possibilities for the distribution of the mixture components are:

<ul><li><a href="Binomial_distribution" title="Binomial distribution">Binomial distribution</a>, for the number of "positive occurrences" (e.g., successes, yes votes, etc.) given a fixed number of total occurrences</li>

<li><a href="Multinomial_distribution" title="Multinomial distribution">Multinomial distribution</a>, similar to the binomial distribution, but for counts of multi-way occurrences (e.g., yes/no/maybe in a survey)</li>

<li><a href="Negative_binomial_distribution" title="Negative binomial distribution">Negative binomial distribution</a>, for binomial-type observations but where the quantity of interest is the number of failures before a given number of successes occurs</li>

<li><a href="Poisson_distribution" title="Poisson distribution">Poisson distribution</a>, for the number of occurrences of an event in a given period of time, for an event that is characterized by a fixed rate of occurrence</li>

<li><a href="Exponential_distribution" title="Exponential distribution">Exponential distribution</a>, for the time before the next event occurs, for an event that is characterized by a fixed rate of occurrence</li>

<li><a href="Log-normal_distribution" title="Log-normal distribution">Log-normal distribution</a>, for positive real numbers that are assumed to grow exponentially, such as incomes or prices</li>

<li><a href="Multivariate_normal_distribution" title="Multivariate normal distribution">Multivariate normal distribution</a> (aka multivariate Gaussian distribution), for vectors of correlated outcomes that are individually Gaussian-distributed</li>

<li><a href="Multivariate_t-distribution" title="Multivariate t-distribution">Multivariate Student's t-distribution</a>, for vectors of heavy-tailed correlated outcomes<a href="#cite_note-2">[2]</a></li>

<li>A vector of <a href="Bernoulli_distribution" title="Bernoulli distribution">Bernoulli</a>-distributed values, corresponding, e.g., to a black-and-white image, with each value representing a pixel; see the handwriting-recognition example below</li></ul>

<div class="mw-heading mw-heading3"><h3 id="Specific_examples">Specific examples</h3></div>

<div class="mw-heading mw-heading4"><h4 id="Gaussian_mixture_model">Gaussian mixture model</h4></div>

A typical non-Bayesian <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian</a> mixture model looks like this:

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>mean of component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>variance of component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

</mrow>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N}&=&{\text{as above}}\\\theta _{i=1\dots K}&=&\{\mu _{i=1\dots K},\sigma _{i=1\dots K}^{2}\}\\\mu _{i=1\dots K}&=&{\text{mean of component }}i\\\sigma _{i=1\dots K}^{2}&=&{\text{variance of component }}i\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\mathcal {N}}(\mu _{z_{i}},\sigma _{z_{i}}^{2})\end{array}}}</annotation>

</semantics>

</math><img src="./6f54869aea02dad65150187b4cfb0bf6687dc8eb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -12.511ex; margin-bottom: -0.327ex; width:47.214ex; height:26.843ex;" alt="{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N}&=&{\text{as above}}\\\theta _{i=1\dots K}&=&\{\mu _{i=1\dots K},\sigma _{i=1\dots K}^{2}\}\\\mu _{i=1\dots K}&=&{\text{mean of component }}i\\\sigma _{i=1\dots K}^{2}&=&{\text{variance of component }}i\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\mathcal {N}}(\mu _{z_{i}},\sigma _{z_{i}}^{2})\end{array}}}" loading="lazy"></dd></dl>

A Bayesian version of a <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian</a> mixture model is as follows:

<mtr>

<mtd>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>as above</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>mean of component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>variance of component </mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>shared hyperparameters</mtext>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mtd>

<mtd>

</mrow>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

</mrow>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N}&=&{\text{as above}}\\\theta _{i=1\dots K}&=&\{\mu _{i=1\dots K},\sigma _{i=1\dots K}^{2}\}\\\mu _{i=1\dots K}&=&{\text{mean of component }}i\\\sigma _{i=1\dots K}^{2}&=&{\text{variance of component }}i\\\mu _{0},\lambda ,\nu ,\sigma _{0}^{2}&=&{\text{shared hyperparameters}}\\\mu _{i=1\dots K}&\sim &{\mathcal {N}}(\mu _{0},\lambda \sigma _{i}^{2})\\\sigma _{i=1\dots K}^{2}&\sim &\operatorname {Inverse-Gamma} (\nu ,\sigma _{0}^{2})\\{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\mathcal {N}}(\mu _{z_{i}},\sigma _{z_{i}}^{2})\end{array}}}</annotation>

</semantics>

</math><img src="./bc2297cfa94e7a3fdabb337a37719131632b7183.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -19.838ex; width:48.638ex; height:40.843ex;" alt="{\displaystyle {\begin{array}{lcl}K,N&=&{\text{as above}}\\\phi _{i=1\dots K},{\boldsymbol {\phi }}&=&{\text{as above}}\\z_{i=1\dots N},x_{i=1\dots N}&=&{\text{as above}}\\\theta _{i=1\dots K}&=&\{\mu _{i=1\dots K},\sigma _{i=1\dots K}^{2}\}\\\mu _{i=1\dots K}&=&{\text{mean of component }}i\\\sigma _{i=1\dots K}^{2}&=&{\text{variance of component }}i\\\mu _{0},\lambda ,\nu ,\sigma _{0}^{2}&=&{\text{shared hyperparameters}}\\\mu _{i=1\dots K}&\sim &{\mathcal {N}}(\mu _{0},\lambda \sigma _{i}^{2})\\\sigma _{i=1\dots K}^{2}&\sim &\operatorname {Inverse-Gamma} (\nu ,\sigma _{0}^{2})\\{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\mathcal {N}}(\mu _{z_{i}},\sigma _{z_{i}}^{2})\end{array}}}" loading="lazy"><math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle }">

</mrow>

<annotation encoding="application/x-tex">{\displaystyle }</annotation>

</semantics>

</math><img src="./df4dcd61276328f7c7ec5bdc399b6e11114a2c68.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.171ex; width:0; height:0.343ex;" alt="{\displaystyle }" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading4"><h4 id="Multivariate_Gaussian_mixture_model">Multivariate Gaussian mixture model</h4></div>

A Bayesian Gaussian mixture model is commonly extended to fit a vector of unknown parameters (denoted in bold), or multivariate normal distributions. In a multivariate distribution (i.e. one modelling a vector <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {x}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {x}}}</annotation>

</semantics>

</math><img src="./606b7680d510560a505937143775ea80fa958051.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.532ex; height:1.676ex;" alt="{\displaystyle {\boldsymbol {x}}}" loading="lazy"> with N random variables) one may model a vector of parameters (such as several observations of a signal or patches within an image) using a Gaussian mixture model prior distribution on the vector of estimates given by

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p({\boldsymbol {\theta }})=\sum _{i=1}^{K}\phi _{i}{\mathcal {N}}({\boldsymbol {\mu }}_{i},{\boldsymbol {\Sigma }}_{i})}</annotation>

</semantics>

</math>

where the ith vector component is characterized by normal distributions with weights <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \phi _{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \phi _{i}}</annotation>

</semantics>

</math><img src="./0182dbf29b54844c92fd9b0311778a02a38398ec.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.185ex; height:2.509ex;" alt="{\displaystyle \phi _{i}}" loading="lazy">, means <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\mu }}_{i}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\mu }}_{i}}</annotation>

</semantics>

</math><img src="./1fb8eff83748a7c926b7fb0774c469d6dd949a5d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:2.445ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\mu }}_{i}}" loading="lazy"> and covariance matrices <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\Sigma }}_{i}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\Sigma }}_{i}}</annotation>

</semantics>

</math><img src="./97c8886bec170246b653eb2dd53699a27e77514b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.731ex; height:2.509ex;" alt="{\displaystyle {\boldsymbol {\Sigma }}_{i}}" loading="lazy">. To incorporate this prior into a Bayesian estimation, the prior is multiplied with the known distribution <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p({\boldsymbol {x|\theta }})}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p({\boldsymbol {x|\theta }})}</annotation>

</semantics>

</math><img src="./1234f137a9961b079f0347666d7b8e94d64d7385.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:6.553ex; height:2.843ex;" alt="{\displaystyle p({\boldsymbol {x|\theta }})}" loading="lazy"> of the data <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {x}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {x}}}</annotation>

</semantics>

</math><img src="./606b7680d510560a505937143775ea80fa958051.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.532ex; height:1.676ex;" alt="{\displaystyle {\boldsymbol {x}}}" loading="lazy"> conditioned on the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> to be estimated. With this formulation, the <a href="Posterior_probability" title="Posterior probability">posterior distribution</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p({\boldsymbol {\theta |x}})}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p({\boldsymbol {\theta |x}})}</annotation>

</semantics>

</math><img src="./69054073f2eb297b74a9260420160378e0f63ddb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:6.553ex; height:2.843ex;" alt="{\displaystyle p({\boldsymbol {\theta |x}})}" loading="lazy"> is also a Gaussian mixture model of the form

</mrow>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p({\boldsymbol {\theta |x}})=\sum _{i=1}^{K}{\tilde {\phi }}_{i}{\mathcal {N}}({\boldsymbol {{\tilde {\mu }}_{i}}},{\boldsymbol {\tilde {\Sigma }}}_{i})}</annotation>

</semantics>

</math>

with new parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\tilde {\phi }}_{i},{\boldsymbol {\tilde {\mu }}}_{i}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\tilde {\phi }}_{i},{\boldsymbol {\tilde {\mu }}}_{i}}</annotation>

</semantics>

</math><img src="./a5143f992e61653066b68789009f1ce3da2cd8f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.746ex; height:3.176ex;" alt="{\displaystyle {\tilde {\phi }}_{i},{\boldsymbol {\tilde {\mu }}}_{i}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\tilde {\Sigma }}}_{i}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\tilde {\Sigma }}}_{i}}</annotation>

</semantics>

</math><img src="./18d763c1c85a49ac7a1fedc13882f1070e4c6e30.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.731ex; height:3.176ex;" alt="{\displaystyle {\boldsymbol {\tilde {\Sigma }}}_{i}}" loading="lazy"> that are updated using the <a href="Expectation-maximization_algorithm" class="mw-redirect" title="Expectation-maximization algorithm">EM algorithm</a>.

<a href="#cite_note-3">[3]</a> Although EM-based parameter updates are well-established, providing the initial estimates for these parameters is currently an area of active research. Note that this formulation yields a closed-form solution to the complete posterior distribution. Estimations of the random variable <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> may be obtained via one of several estimators, such as the mean or maximum of the posterior distribution.

Such distributions are useful for assuming patch-wise shapes of images and clusters, for example. In the case of image representation, each Gaussian may be tilted, expanded, and warped according to the covariance matrices <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\Sigma }}_{i}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\Sigma }}_{i}}</annotation>

</semantics>

</math><img src="./97c8886bec170246b653eb2dd53699a27e77514b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.731ex; height:2.509ex;" alt="{\displaystyle {\boldsymbol {\Sigma }}_{i}}" loading="lazy">. One Gaussian distribution of the set is fit to each patch (usually of size 8×8 pixels) in the image. Notably, any distribution of points around a cluster (see <a href="K-means_clustering" title="K-means clustering">k-means</a>) may be accurately given enough Gaussian components, but scarcely over K=20 components are needed to accurately model a given image distribution or cluster of data.

<div class="mw-heading mw-heading4"><h4 id="Categorical_mixture_model">Categorical mixture model</h4></div>

A typical non-Bayesian mixture model with <a href="Categorical_distribution" title="Categorical distribution">categorical</a> observations looks like this:

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K,N:}</annotation>

</semantics>

</math><img src="./99458f5e86c26e5116bbd38b5785dd3281ae08e7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.456ex; height:2.509ex;" alt="{\displaystyle K,N:}" loading="lazy"> as above</li>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \phi _{i=1\dots K},{\boldsymbol {\phi }}:}</annotation>

</semantics>

</math><img src="./2a1f5a04a02e7da2f8822778b7484ad4322b555b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.653ex; height:2.509ex;" alt="{\displaystyle \phi _{i=1\dots K},{\boldsymbol {\phi }}:}" loading="lazy"> as above</li>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle z_{i=1\dots N},x_{i=1\dots N}:}</annotation>

</semantics>

</math><img src="./30917979248e96e82a84c435cc04612c1b224c31.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:17.307ex; height:2.009ex;" alt="{\displaystyle z_{i=1\dots N},x_{i=1\dots N}:}" loading="lazy"> as above</li>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V:}</annotation>

</semantics>

</math><img src="./a6bf66ce0eadf19346f6c9b07f4dfab82b9d77dc.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.079ex; height:2.176ex;" alt="{\displaystyle V:}" loading="lazy"> dimension of categorical observations, e.g., size of word vocabulary</li>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta _{i=1\dots K,j=1\dots V}:}</annotation>

</semantics>

</math><img src="./7bef275f5676205a98adc208bb4b70cf8f8dc94d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:15.094ex; height:2.843ex;" alt="{\displaystyle \theta _{i=1\dots K,j=1\dots V}:}" loading="lazy"> probability for component <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle i}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle i}</annotation>

</semantics>

</math><img src="./add78d8608ad86e54951b8c8bd6c8d8416533d20.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.802ex; height:2.176ex;" alt="{\displaystyle i}" loading="lazy"> of observing item <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle j}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle j}</annotation>

</semantics>

</math><img src="./2f461e54f5c093e92a55547b9764291390f0b5d0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; margin-left: -0.027ex; width:0.985ex; height:2.509ex;" alt="{\displaystyle j}" loading="lazy"></li>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}_{i=1\dots K}:}</annotation>

</semantics>

</math><img src="./0ee85fdb36bef2ee33bd0d093c84c548b125eb32.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:8.885ex; height:2.509ex;" alt="{\displaystyle {\boldsymbol {\theta }}_{i=1\dots K}:}" loading="lazy"> vector of dimension <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle V,}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V,}</annotation>

</semantics>

</math><img src="./ace9595e3ce66fdec7e9d30202626accd676b11e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.434ex; height:2.509ex;" alt="{\displaystyle V,}" loading="lazy"> composed of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \theta _{i,1\dots V};}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta _{i,1\dots V};}</annotation>

</semantics>

</math><img src="./4ddea5657393f6bcf15a05e1b57cdb2359d10dea.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:7.006ex; height:2.843ex;" alt="{\displaystyle \theta _{i,1\dots V};}" loading="lazy"> must sum to 1</li></ul>

The random variables:

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>Categorical</mtext>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\text{Categorical}}({\boldsymbol {\theta }}_{z_{i}})\end{array}}}</annotation>

</semantics>

</math><img src="./b5f72f9dc7972cb21c5e9c07c4326d4dc8e63004.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:31.06ex; height:6.509ex;" alt="{\displaystyle {\begin{array}{lcl}z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\text{Categorical}}({\boldsymbol {\theta }}_{z_{i}})\end{array}}}" loading="lazy"></dd></dl>

A typical Bayesian mixture model with <a href="Categorical_distribution" title="Categorical distribution">categorical</a> observations looks like this:

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K,N:}</annotation>

</semantics>

</math><img src="./99458f5e86c26e5116bbd38b5785dd3281ae08e7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:6.456ex; height:2.509ex;" alt="{\displaystyle K,N:}" loading="lazy"> as above</li>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \phi _{i=1\dots K},{\boldsymbol {\phi }}:}</annotation>

</semantics>

</math><img src="./2a1f5a04a02e7da2f8822778b7484ad4322b555b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.653ex; height:2.509ex;" alt="{\displaystyle \phi _{i=1\dots K},{\boldsymbol {\phi }}:}" loading="lazy"> as above</li>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle z_{i=1\dots N},x_{i=1\dots N}:}</annotation>

</semantics>

</math><img src="./30917979248e96e82a84c435cc04612c1b224c31.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:17.307ex; height:2.009ex;" alt="{\displaystyle z_{i=1\dots N},x_{i=1\dots N}:}" loading="lazy"> as above</li>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V:}</annotation>

</semantics>

</math><img src="./a6bf66ce0eadf19346f6c9b07f4dfab82b9d77dc.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.079ex; height:2.176ex;" alt="{\displaystyle V:}" loading="lazy"> dimension of categorical observations, e.g., size of word vocabulary</li>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta _{i=1\dots K,j=1\dots V}:}</annotation>

</semantics>

</math><img src="./7bef275f5676205a98adc208bb4b70cf8f8dc94d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:15.094ex; height:2.843ex;" alt="{\displaystyle \theta _{i=1\dots K,j=1\dots V}:}" loading="lazy"> probability for component <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle i}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle i}</annotation>

</semantics>

</math><img src="./add78d8608ad86e54951b8c8bd6c8d8416533d20.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.802ex; height:2.176ex;" alt="{\displaystyle i}" loading="lazy"> of observing item <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle j}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle j}</annotation>

</semantics>

</math><img src="./2f461e54f5c093e92a55547b9764291390f0b5d0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; margin-left: -0.027ex; width:0.985ex; height:2.509ex;" alt="{\displaystyle j}" loading="lazy"></li>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}_{i=1\dots K}:}</annotation>

</semantics>

</math><img src="./0ee85fdb36bef2ee33bd0d093c84c548b125eb32.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:8.885ex; height:2.509ex;" alt="{\displaystyle {\boldsymbol {\theta }}_{i=1\dots K}:}" loading="lazy"> vector of dimension <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle V,}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle V,}</annotation>

</semantics>

</math><img src="./ace9595e3ce66fdec7e9d30202626accd676b11e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.434ex; height:2.509ex;" alt="{\displaystyle V,}" loading="lazy"> composed of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \theta _{i,1\dots V};}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta _{i,1\dots V};}</annotation>

</semantics>

</math><img src="./4ddea5657393f6bcf15a05e1b57cdb2359d10dea.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:7.006ex; height:2.843ex;" alt="{\displaystyle \theta _{i,1\dots V};}" loading="lazy"> must sum to 1</li>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \alpha :}</annotation>

</semantics>

</math><img src="./3dc633c1cfa93fe91f681273fcebb66d206e66bd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.78ex; height:1.676ex;" alt="{\displaystyle \alpha :}" loading="lazy"> shared concentration hyperparameter of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> for each component</li>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \beta :}</annotation>

</semantics>

</math><img src="./97cc108ed0f06f749a1bb2fa78cfa626a7d5eb28.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.624ex; height:2.509ex;" alt="{\displaystyle \beta :}" loading="lazy"> concentration hyperparameter of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\phi }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\phi }}}</annotation>

</semantics>

</math><img src="./2162da9f005868f9129e85261efe96e619fffdab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.655ex; height:2.509ex;" alt="{\displaystyle {\boldsymbol {\phi }}}" loading="lazy"></li></ul>

The random variables:

<mtr>

<mtd>

</mrow>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<msub>

<mtext>Symmetric-Dirichlet</mtext>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mi>Categorical</mi>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

</mtd>

<mtd>

<mtext>Categorical</mtext>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{array}{lcl}{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\{\boldsymbol {\theta }}_{i=1\dots K}&\sim &{\text{Symmetric-Dirichlet}}_{V}(\alpha )\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\text{Categorical}}({\boldsymbol {\theta }}_{z_{i}})\end{array}}}</annotation>

</semantics>

</math><img src="./4b703bb83590855b47bc8b9e4cbc8df0d20d4a6d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -6.005ex; width:40.238ex; height:13.176ex;" alt="{\displaystyle {\begin{array}{lcl}{\boldsymbol {\phi }}&\sim &\operatorname {Symmetric-Dirichlet} _{K}(\beta )\\{\boldsymbol {\theta }}_{i=1\dots K}&\sim &{\text{Symmetric-Dirichlet}}_{V}(\alpha )\\z_{i=1\dots N}&\sim &\operatorname {Categorical} ({\boldsymbol {\phi }})\\x_{i=1\dots N}&\sim &{\text{Categorical}}({\boldsymbol {\theta }}_{z_{i}})\end{array}}}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading2"><h2 id="Examples">Examples</h2></div>

<div class="mw-heading mw-heading3"><h3 id="A_financial_model">A financial model</h3></div>

Financial returns often behave differently in normal situations and during crisis times. A mixture model<a href="#cite_note-4">[4]</a> for return data seems reasonable. Sometimes the model used is a <a href="Jump-diffusion_model" class="mw-redirect" title="Jump-diffusion model">jump-diffusion model</a>, or as a mixture of two normal distributions. See <a href="Financial_economics#Challenges_and_criticism" title="Financial economics">Financial economics § Challenges and criticism</a> and <a href="Financial_risk_management#Banking" title="Financial risk management">Financial risk management § Banking</a> for further context.

<div class="mw-heading mw-heading3"><h3 id="House_prices">House prices</h3></div>

Assume that we observe the prices of N different houses. Different types of houses in different neighborhoods will have vastly different prices, but the price of a particular type of house in a particular neighborhood (e.g., three-bedroom house in moderately upscale neighborhood) will tend to cluster fairly closely around the mean. One possible model of such prices would be to assume that the prices are accurately described by a mixture model with K different components, each distributed as a <a href="Normal_distribution" title="Normal distribution">normal distribution</a> with unknown mean and variance, with each component specifying a particular combination of house type/neighborhood. Fitting this model to observed prices, e.g., using the <a href="Expectation-maximization_algorithm" class="mw-redirect" title="Expectation-maximization algorithm">expectation-maximization algorithm</a>, would tend to cluster the prices according to house type/neighborhood and reveal the spread of prices in each type/neighborhood. (Note that for values such as prices or incomes that are guaranteed to be positive and which tend to grow <a href="Exponential_growth" title="Exponential growth">exponentially</a>, a <a href="Log-normal_distribution" title="Log-normal distribution">log-normal distribution</a> might actually be a better model than a normal distribution.)

<div class="mw-heading mw-heading3"><h3 id="Topics_in_a_document">Topics in a document</h3></div>

Assume that a document is composed of N different words from a total vocabulary of size V, where each word corresponds to one of K possible topics. The distribution of such words could be modelled as a mixture of K different V-dimensional <a href="Categorical_distribution" title="Categorical distribution">categorical distributions</a>. A model of this sort is commonly termed a <a href="Topic_model" title="Topic model">topic model</a>. Note that <a href="Expectation_maximization" class="mw-redirect" title="Expectation maximization">expectation maximization</a> applied to such a model will typically fail to produce realistic results, due (among other things) to the <a href="Overfitting" title="Overfitting">excessive number of parameters</a>. Some sorts of additional assumptions are typically necessary to get good results. Typically two sorts of additional components are added to the model:

<ol><li>A <a href="Prior_distribution" class="mw-redirect" title="Prior distribution">prior distribution</a> is placed over the parameters describing the topic distributions, using a <a href="Dirichlet_distribution" title="Dirichlet distribution">Dirichlet distribution</a> with a <a href="Concentration_parameter" title="Concentration parameter">concentration parameter</a> that is set significantly below 1, so as to encourage sparse distributions (where only a small number of words have significantly non-zero probabilities).</li>

<li>Some sort of additional constraint is placed over the topic identities of words, to take advantage of natural clustering.

<ul><li>For example, a <a href="Markov_chain" title="Markov chain">Markov chain</a> could be placed on the topic identities (i.e., the latent variables specifying the mixture component of each observation), corresponding to the fact that nearby words belong to similar topics. (This results in a <a href="Hidden_Markov_model" title="Hidden Markov model">hidden Markov model</a>, specifically one where a <a href="Prior_distribution" class="mw-redirect" title="Prior distribution">prior distribution</a> is placed over state transitions that favors transitions that stay in the same state.)</li>

<li>Another possibility is the <a href="Latent_Dirichlet_allocation" title="Latent Dirichlet allocation">latent Dirichlet allocation</a> model, which divides up the words into D different documents and assumes that in each document only a small number of topics occur with any frequency.</li></ul></li></ol>

<div class="mw-heading mw-heading3"><h3 id="Handwriting_recognition">Handwriting recognition</h3></div>

The following example is based on an example in <a href="Christopher_M._Bishop" class="mw-redirect" title="Christopher M. Bishop">Christopher M. Bishop</a>, Pattern Recognition and Machine Learning.<a href="#cite_note-5">[5]</a>

Imagine that we are given an N×N black-and-white image that is known to be a scan of a hand-written digit between 0 and 9, but we don't know which digit is written. We can create a mixture model with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle K=10}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K=10}</annotation>

</semantics>

</math><img src="./4f5caa2945733b3b3207347fdf814714cefce3fb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:7.489ex; height:2.176ex;" alt="{\displaystyle K=10}" loading="lazy"> different components, where each component is a vector of size <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle N^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle N^{2}}</annotation>

</semantics>

</math><img src="./fe131b76af8a2bc86e01b14a7ba843db69c1a164.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.177ex; height:2.676ex;" alt="{\displaystyle N^{2}}" loading="lazy"> of <a href="Bernoulli_distribution" title="Bernoulli distribution">Bernoulli distributions</a> (one per pixel). Such a model can be trained with the <a href="Expectation-maximization_algorithm" class="mw-redirect" title="Expectation-maximization algorithm">expectation-maximization algorithm</a> on an unlabeled set of hand-written digits, and will effectively cluster the images according to the digit being written. The same model could then be used to recognize the digit of another image simply by holding the parameters constant, computing the probability of the new image for each possible digit (a trivial calculation), and returning the digit that generated the highest probability.

<div class="mw-heading mw-heading3"><h3 id="Assessing_projectile_accuracy_(a.k.a._circular_error_probable,_CEP)">Assessing projectile accuracy (a.k.a. circular error probable, CEP)</h3></div>

Mixture models apply in the problem of directing multiple projectiles at a target (as in air, land, or sea defense applications), where the physical and/or statistical characteristics of the projectiles differ within the multiple projectiles. An example might be shots from multiple munitions types or shots from multiple locations directed at one target. The combination of projectile types may be characterized as a Gaussian mixture model.<a href="#cite_note-6">[6]</a> Further, a well-known measure of accuracy for a group of projectiles is the <a href="Circular_error_probable" title="Circular error probable">circular error probable</a> (CEP), which is the number R such that, on average, half of the group of projectiles falls within the circle of radius R about the target point. The mixture model can be used to determine (or estimate) the value R. The mixture model properly captures the different types of projectiles.

<div class="mw-heading mw-heading3"><h3 id="Direct_and_indirect_applications">Direct and indirect applications</h3></div>

The financial example above is one direct application of the mixture model, a situation in which we assume an underlying mechanism so that each observation belongs to one of some number of different sources or categories. This underlying mechanism may or may not, however, be observable. In this form of mixture, each of the sources is described by a component probability density function, and its mixture weight is the probability that an observation comes from this component.

In an indirect application of the mixture model we do not assume such a mechanism. The mixture model is simply used for its mathematical flexibilities. For example, a mixture of two <a href="Normal_distribution" title="Normal distribution">normal distributions</a> with different means may result in a density with two <a href="Mode_(statistics)" title="Mode (statistics)">modes</a>, which is not modeled by standard parametric distributions. Another example is given by the possibility of mixture distributions to model fatter tails than the basic Gaussian ones, so as to be a candidate for modeling more extreme events.

<div class="mw-heading mw-heading3"><h3 id="Predictive_Maintenance">Predictive Maintenance</h3></div>

The mixture model-based clustering is also predominantly used in identifying the state of the machine in <a href="Predictive_maintenance" title="Predictive maintenance">predictive maintenance</a>. Density plots are used to analyze the density of high dimensional features. If multi-model densities are observed, then it is assumed that a finite set of densities are formed by a finite set of normal mixtures. A multivariate Gaussian mixture model is used to cluster the feature data into k number of groups where k represents each state of the machine. The machine state can be a normal state, power off state, or faulty state.<a href="#cite_note-7">[7]</a> Each formed cluster can be diagnosed using techniques such as spectral analysis. In the recent years, this has also been widely used in other areas such as early fault detection.<a href="#cite_note-8">[8]</a>

<div class="mw-heading mw-heading3"><h3 id="Fuzzy_image_segmentation">Fuzzy image segmentation</h3></div>

In image processing and computer vision, traditional <a href="Image_segmentation" title="Image segmentation">image segmentation</a> models often assign to one <a href="Pixel" title="Pixel">pixel</a> only one exclusive pattern. In fuzzy or soft segmentation, any pattern can have certain "ownership" over any single pixel. If the patterns are Gaussian, fuzzy segmentation naturally results in Gaussian mixtures. Combined with other analytic or geometric tools (e.g., phase transitions over diffusive boundaries), such spatially regularized mixture models could lead to more realistic and computationally efficient segmentation methods.<a href="#cite_note-9">[9]</a>

<div class="mw-heading mw-heading3"><h3 id="Point_set_registration">Point set registration</h3></div>

Probabilistic mixture models such as <a href="Gaussian_mixture_model" class="mw-redirect" title="Gaussian mixture model">Gaussian mixture models</a> (GMM) are used to resolve <a href="Point_set_registration" class="mw-redirect" title="Point set registration">point set registration</a> problems in image processing and computer vision fields. For pair-wise <a href="Point_set_registration" class="mw-redirect" title="Point set registration">point set registration</a>, one point set is regarded as the centroids of mixture models, and the other point set is regarded as data points (observations). State-of-the-art methods are e.g. <a href="Point_set_registration" class="mw-redirect" title="Point set registration">coherent point drift</a> (CPD)<a href="#cite_note-10">[10]</a>

and <a href="Student's_t-distribution" title="Student's t-distribution">Student's t-distribution</a> mixture models (TMM).<a href="#cite_note-11">[11]</a>

The result of recent research demonstrate the superiority of hybrid mixture models<a href="#cite_note-12">[12]</a>

(e.g. combining Student's t-distribution and Watson distribution/<a href="Bingham_distribution" title="Bingham distribution">Bingham distribution</a> to model spatial positions and axes orientations separately) compare to CPD and TMM, in terms of inherent robustness, accuracy and discriminative capacity.

<div class="mw-heading mw-heading2"><h2 id="Identifiability">Identifiability</h2></div>

Identifiability refers to the existence of a unique characterization for any one of the models in the class (family) being considered. Estimation procedures may not be well-defined and asymptotic theory may not hold if a model is not identifiable.

<div class="mw-heading mw-heading3"><h3 id="Example">Example</h3></div>

Let J be the class of all binomial distributions with n = 2. Then a mixture of two members of J would have

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

</mrow>

</mrow>

<msup>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</msub>

</mtd>

<mtd>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}p_{0}&=\pi {\left(1-\theta _{1}\right)}^{2}+\left(1-\pi \right){\left(1-\theta _{2}\right)}^{2}\\[1ex]p_{1}&=2\pi \theta _{1}\left(1-\theta _{1}\right)+2\left(1-\pi \right)\theta _{2}\left(1-\theta _{2}\right)\end{aligned}}}</annotation>

</semantics>

</math>

and p2 = 1 − p0 − p1. Clearly, given p0 and p1, it is not possible to determine the above mixture model uniquely, as there are three parameters (π, θ1, θ2) to be determined.

<div class="mw-heading mw-heading3"><h3 id="Definition">Definition</h3></div>

Consider a mixture of parametric distributions of the same class. Let

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle J=\{f(\cdot ;\theta ):\theta \in \Omega \}}</annotation>

</semantics>

</math>

be the class of all component distributions. Then the <a href="Convex_hull" title="Convex hull">convex hull</a> K of J defines the class of all finite mixture of distributions in J:

<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle K=\left\{p(\cdot ):p(\cdot )=\sum _{i=1}^{n}a_{i}f_{i}(\cdot ;\theta _{i}),a_{i}>0,\sum _{i=1}^{n}a_{i}=1,f_{i}(\cdot ;\theta _{i})\in J\ \forall i,n\right\}}">

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle K=\left\{p(\cdot ):p(\cdot )=\sum _{i=1}^{n}a_{i}f_{i}(\cdot ;\theta _{i}),a_{i}>0,\sum _{i=1}^{n}a_{i}=1,f_{i}(\cdot ;\theta _{i})\in J\ \forall i,n\right\}}</annotation>

</semantics>

</math>

K is said to be identifiable if all its members are unique, that is, given two members p and p′ in K, being mixtures of k distributions and k′ distributions respectively in J, we have p = p′ if and only if, first of all, k = k′ and secondly we can reorder the summations such that ai = ai′ and fi = fi′ for all i.

<div class="mw-heading mw-heading2"><h2 id="Parameter_estimation_and_system_identification">Parameter estimation and system identification</h2></div>

Parametric mixture models are often used when we know the distribution Y and we can sample from X, but we would like to determine the ai and θi values. Such situations can arise in studies in which we sample from a population that is composed of several distinct subpopulations.

It is common to think of probability mixture modeling as a missing data problem. One way to understand this is to assume that the data points under consideration have "membership" in one of the distributions we are using to model the data. When we start, this membership is unknown, or missing. The job of estimation is to devise appropriate parameters for the model functions we choose, with the connection to the data points being represented as their membership in the individual model distributions.

A variety of approaches to the problem of mixture decomposition have been proposed, many of which focus on maximum likelihood methods such as <a href="Expectation_maximization" class="mw-redirect" title="Expectation maximization">expectation maximization</a> (EM) or maximum a posteriori estimation (MAP). Generally these methods consider separately the questions of system identification and parameter estimation; methods to determine the number and functional form of components within a mixture are distinguished from methods to estimate the corresponding parameter values. Some notable departures are the graphical methods as outlined in Tarter and Lock<a href="#cite_note-tart-13">[13]</a> and more recently <a href="Minimum_message_length" title="Minimum message length">minimum message length</a> (MML) techniques such as Figueiredo and Jain<a href="#cite_note-Jain-14">[14]</a> and to some extent the moment matching pattern analysis routines suggested by McWilliam and Loh (2009).<a href="#cite_note-mcwilli-15">[15]</a>

<div class="mw-heading mw-heading3"><h3 id="Expectation_maximization_(EM)">Expectation maximization (EM)</h3></div>

<a href="Expectation-maximization_algorithm" class="mw-redirect" title="Expectation-maximization algorithm">Expectation maximization</a> (EM) is seemingly the most popular technique used to determine the parameters of a mixture with an a priori given number of components. This is a particular way of implementing <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> estimation for this problem. EM is of particular appeal for finite normal mixtures where closed-form expressions are possible such as in the following iterative algorithm by Dempster et al. (1977)<a href="#cite_note-dempster1977-16">[16]</a>

</mrow>

</mrow>

</msubsup>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{s}^{(j+1)}={\frac {1}{N}}\sum _{t=1}^{N}h_{s}^{(j)}(t)}</annotation>

</semantics>

</math><img src="./e3b3594cc4e64e49c4ba357e9669fb44c3076af5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:22.258ex; height:7.343ex;" alt="{\displaystyle w_{s}^{(j+1)}={\frac {1}{N}}\sum _{t=1}^{N}h_{s}^{(j)}(t)}" loading="lazy"></dd>

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu _{s}^{(j+1)}={\frac {\sum _{t=1}^{N}h_{s}^{(j)}(t)x^{(t)}}{\sum _{t=1}^{N}h_{s}^{(j)}(t)}}}</annotation>

</semantics>

</math><img src="./3aeda49e95d7437ef5798fe9e1edd14ffaa404a0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:25.005ex; height:8.176ex;" alt="{\displaystyle \mu _{s}^{(j+1)}={\frac {\sum _{t=1}^{N}h_{s}^{(j)}(t)x^{(t)}}{\sum _{t=1}^{N}h_{s}^{(j)}(t)}}}" loading="lazy"></dd>

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Sigma _{s}^{(j+1)}={\frac {\sum _{t=1}^{N}h_{s}^{(j)}(t)[x^{(t)}-\mu _{s}^{(j+1)}][x^{(t)}-\mu _{s}^{(j+1)}]^{\top }}{\sum _{t=1}^{N}h_{s}^{(j)}(t)}}}</annotation>

</semantics>

</math><img src="./0198bb195ea56a45f53c21b0c1f4322c126063d4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:49.878ex; height:8.176ex;" alt="{\displaystyle \Sigma _{s}^{(j+1)}={\frac {\sum _{t=1}^{N}h_{s}^{(j)}(t)[x^{(t)}-\mu _{s}^{(j+1)}][x^{(t)}-\mu _{s}^{(j+1)}]^{\top }}{\sum _{t=1}^{N}h_{s}^{(j)}(t)}}}" loading="lazy"></dd></dl>

with the posterior probabilities

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle h_{s}^{(j)}(t)={\frac {w_{s}^{(j)}p_{s}(x^{(t)};\mu _{s}^{(j)},\Sigma _{s}^{(j)})}{\sum _{i=1}^{n}w_{i}^{(j)}p_{i}(x^{(t)};\mu _{i}^{(j)},\Sigma _{i}^{(j)})}}.}</annotation>

</semantics>

</math><img src="./6348a43581944bc3f4c527fd3aea50ec25317e47.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:37.092ex; height:8.009ex;" alt="{\displaystyle h_{s}^{(j)}(t)={\frac {w_{s}^{(j)}p_{s}(x^{(t)};\mu _{s}^{(j)},\Sigma _{s}^{(j)})}{\sum _{i=1}^{n}w_{i}^{(j)}p_{i}(x^{(t)};\mu _{i}^{(j)},\Sigma _{i}^{(j)})}}.}" loading="lazy"></dd></dl>

Thus on the basis of the current estimate for the parameters, the <a href="Conditional_probability" title="Conditional probability">conditional probability</a> for a given observation x(t) being generated from state s is determined for each t = 1, …, N ; N being the sample size. The parameters are then updated such that the new component weights correspond to the average conditional probability and each component mean and covariance is the component specific weighted average of the mean and covariance of the entire sample.

Dempster<a href="#cite_note-dempster1977-16">[16]</a> also showed that each successive EM iteration will not decrease the likelihood, a property not shared by other gradient based maximization techniques. Moreover, EM naturally embeds within it constraints on the probability vector, and for sufficiently large sample sizes positive definiteness of the covariance iterates. This is a key advantage since explicitly constrained methods incur extra computational costs to check and maintain appropriate values. Theoretically EM is a first-order algorithm and as such converges slowly to a fixed-point solution. Redner and Walker (1984) make this point arguing in favour of superlinear and second order Newton and quasi-Newton methods and reporting slow convergence in EM on the basis of their empirical tests. They do concede that convergence in likelihood was rapid even if convergence in the parameter values themselves was not. The relative merits of EM and other algorithms vis-à-vis convergence have been discussed in other literature.<a href="#cite_note-XuJordam-17">[17]</a>

Other common objections to the use of EM are that it has a propensity to spuriously identify local maxima, as well as displaying sensitivity to initial values.<a href="#cite_note-McLachlan_2-18">[18]</a><a href="#cite_note-botev2004global-19">[19]</a> One may address these problems by evaluating EM at several initial points in the parameter space but this is computationally costly and other approaches, such as the annealing EM method of Udea and Nakano (1998) (in which the initial components are essentially forced to overlap, providing a less heterogeneous basis for initial guesses), may be preferable.

Figueiredo and Jain<a href="#cite_note-Jain-14">[14]</a> note that convergence to 'meaningless' parameter values obtained at the boundary (where regularity conditions breakdown, e.g., Ghosh and Sen (1985)) is frequently observed when the number of model components exceeds the optimal/true one. On this basis they suggest a unified approach to estimation and identification in which the initial n is chosen to greatly exceed the expected optimal value. Their optimization routine is constructed via a minimum message length (MML) criterion that effectively eliminates a candidate component if there is insufficient information to support it. In this way it is possible to systematize reductions in n and consider estimation and identification jointly.

<div class="mw-heading mw-heading4"><h4 id="The_expectation_step">The expectation step</h4></div>

With initial guesses for the parameters of our mixture model, "partial membership" of each data point in each constituent distribution is computed by calculating <a href="Expectation_value" class="mw-redirect" title="Expectation value">expectation values</a> for the membership variables of each data point. That is, for each data point xj and distribution Yi, the membership value yi, j is:

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y_{i,j}={\frac {a_{i}f_{Y}(x_{j};\theta _{i})}{f_{X}(x_{j})}}.}</annotation>

</semantics>

</math><img src="./35539145d63931b0d991dde2e447573bdbf6f091.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.671ex; width:19.283ex; height:6.509ex;" alt="{\displaystyle y_{i,j}={\frac {a_{i}f_{Y}(x_{j};\theta _{i})}{f_{X}(x_{j})}}.}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading4"><h4 id="The_maximization_step">The maximization step</h4></div>

With expectation values in hand for group membership, plug-in estimates are recomputed for the distribution parameters.

The mixing coefficients ai are the <a href="Arithmetic_mean" title="Arithmetic mean">means</a> of the membership values over the N data points.

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle a_{i}={\frac {1}{N}}\sum _{j=1}^{N}y_{i,j}}</annotation>

</semantics>

</math><img src="./eb54eeccd6253130fbb4ed09048f427d7c045553.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.338ex; width:15.231ex; height:7.676ex;" alt="{\displaystyle a_{i}={\frac {1}{N}}\sum _{j=1}^{N}y_{i,j}}" loading="lazy"></dd></dl>

The component model parameters θi are also calculated by expectation maximization using data points xj that have been weighted using the membership values. For example, if θ is a mean μ

<msub>

</mrow>

</msub>

<mfrac>

<mrow>

</mrow>

</munder>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

<mrow>

</mrow>

</munder>

<msub>

</mrow>

</msub>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mu _{i}={\frac {\sum _{j}y_{i,j}x_{j}}{\sum _{j}y_{i,j}}}.}</annotation>

</semantics>

</math><img src="./7b854c575da4da2c5334f5d67c6c6b911d3e6f3f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:15.847ex; height:7.176ex;" alt="{\displaystyle \mu _{i}={\frac {\sum _{j}y_{i,j}x_{j}}{\sum _{j}y_{i,j}}}.}" loading="lazy"></dd></dl>

With new estimates for ai and the θi's, the expectation step is repeated to recompute new membership values. The entire procedure is repeated until model parameters converge.

<div class="mw-heading mw-heading3"><h3 id="Markov_chain_Monte_Carlo">Markov chain Monte Carlo</h3></div>

As an alternative to the EM algorithm, the mixture model parameters can be deduced using posterior sampling as indicated by <a href="Bayes'_theorem" title="Bayes' theorem">Bayes' theorem</a>. This is still regarded as an incomplete data problem in which membership of data points is the missing data. A two-step iterative procedure known as <a href="Gibbs_sampling" title="Gibbs sampling">Gibbs sampling</a> can be used.

The previous example of a mixture of two <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distributions</a> can demonstrate how the method works. As before, initial guesses of the parameters for the mixture model are made. Instead of computing partial memberships for each elemental distribution, a membership value for each data point is drawn from a <a href="Bernoulli_distribution" title="Bernoulli distribution">Bernoulli distribution</a> (that is, it will be assigned to either the first or the second Gaussian). The Bernoulli parameter θ is determined for each data point on the basis of one of the constituent distributions. Draws from the distribution generate membership associations for each data point. Plug-in estimators can then be used as in the M step of EM to generate a new set of mixture model parameters, and the binomial draw step repeated.

<div class="mw-heading mw-heading3"><h3 id="Moment_matching">Moment matching</h3></div>

The <a href="Method_of_moments_(statistics)" title="Method of moments (statistics)">method of moment matching</a> is one of the oldest techniques for determining the mixture parameters dating back to Karl Pearson's seminal work of 1894.

In this approach the parameters of the mixture are determined such that the composite distribution has moments matching some given value. In many instances extraction of solutions to the moment equations may present non-trivial algebraic or computational problems. Moreover, numerical analysis by Day<a href="#cite_note-day-20">[20]</a> has indicated that such methods may be inefficient compared to EM. Nonetheless, there has been renewed interest in this method, e.g., Craigmile and Titterington (1998) and Wang.<a href="#cite_note-wang-21">[21]</a>

McWilliam and Loh (2009) consider the characterisation of a hyper-cuboid normal mixture <a href="Copula_(statistics)" title="Copula (statistics)">copula</a> in large dimensional systems for which EM would be computationally prohibitive. Here a pattern analysis routine is used to generate multivariate tail-dependencies consistent with a set of univariate and (in some sense) bivariate moments. The performance of this method is then evaluated using equity log-return data with <a href="Kolmogorov%E2%80%93Smirnov" class="mw-redirect" title="Kolmogorov–Smirnov">Kolmogorov–Smirnov</a> test statistics suggesting a good descriptive fit.

<div class="mw-heading mw-heading3"><h3 id="Spectral_method">Spectral method</h3></div>

Some problems in mixture model estimation can be solved using <a href="Spectral_method" title="Spectral method">spectral methods</a>.

In particular it becomes useful if data points xi are points in high-dimensional <a href="Real_coordinate_space" title="Real coordinate space">real space</a>, and the hidden distributions are known to be <a href="Logarithmically_concave_function" title="Logarithmically concave function">log-concave</a> (such as <a href="Gaussian_distribution" class="mw-redirect" title="Gaussian distribution">Gaussian distribution</a> or <a href="Exponential_distribution" title="Exponential distribution">Exponential distribution</a>).

Spectral methods of learning mixture models are based on the use of <a href="Singular_Value_Decomposition" class="mw-redirect" title="Singular Value Decomposition">Singular Value Decomposition</a> of a matrix which contains data points.

The idea is to consider the top k singular vectors, where k is the number of distributions to be learned. The projection

of each data point to a <a href="Linear_subspace" title="Linear subspace">linear subspace</a> spanned by those vectors groups points originating from the same distribution

very close together, while points from different distributions stay far apart.

One distinctive feature of the spectral method is that it allows us to <a href="Mathematical_proof" title="Mathematical proof">prove</a> that if

distributions satisfy certain separation condition (e.g., not too close), then the estimated mixture will be very close to the true one with high probability.

<div class="mw-heading mw-heading3"><h3 id="Graphical_Methods">Graphical Methods</h3></div>

Tarter and Lock<a href="#cite_note-tart-13">[13]</a> describe a graphical approach to mixture identification in which a kernel function is applied to an empirical frequency plot so to reduce intra-component variance. In this way one may more readily identify components having differing means. While this λ-method does not require prior knowledge of the number or functional form of the components its success does rely on the choice of the kernel parameters which to some extent implicitly embeds assumptions about the component structure.

<div class="mw-heading mw-heading3"><h3 id="Other_methods">Other methods</h3></div>

Some of them can even probably learn mixtures of <a href="Heavy-tailed_distribution" title="Heavy-tailed distribution">heavy-tailed distributions</a> including those with

infinite <a href="Variance" title="Variance">variance</a> (see <a href="#Recent_Papers">links to papers</a> below).

In this setting, EM based methods would not work, since the Expectation step would diverge due to presence of

<a href="Outlier" title="Outlier">outliers</a>.

<div class="mw-heading mw-heading3"><h3 id="A_simulation">A simulation</h3></div>

To simulate a sample of size N that is from a mixture of distributions Fi, i=1 to n, with probabilities pi (sum= pi = 1):

<ol><li>Generate N random numbers from a <a href="Categorical_distribution" title="Categorical distribution">categorical distribution</a> of size n and probabilities pi for i= 1= to n. These tell you which of the Fi each of the N values will come from. Denote by mi the quantity of random numbers assigned to the ith category.</li>

<li>For each i, generate mi random numbers from the Fi distribution.</li></ol>

<div class="mw-heading mw-heading2"><h2 id="Extensions">Extensions</h2></div>

In a <a href="Bayesian_inference" title="Bayesian inference">Bayesian setting</a>, additional levels can be added to the <a href="Graphical_model" title="Graphical model">graphical model</a> defining the mixture model. For example, in the common <a href="Latent_Dirichlet_allocation" title="Latent Dirichlet allocation">latent Dirichlet allocation</a> <a href="Topic_model" title="Topic model">topic model</a>, the observations are sets of words drawn from D different documents and the K mixture components represent topics that are shared across documents. Each document has a different set of mixture weights, which specify the topics prevalent in that document. All sets of mixture weights share common <a href="Hyperparameter_(Bayesian_statistics)" title="Hyperparameter (Bayesian statistics)">hyperparameters</a>.

A very common extension is to connect the <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a> defining the mixture component identities into a <a href="Markov_chain" title="Markov chain">Markov chain</a>, instead of assuming that they are <a href="Independent_identically_distributed" class="mw-redirect" title="Independent identically distributed">independent identically distributed</a> random variables. The resulting model is termed a <a href="Hidden_Markov_model" title="Hidden Markov model">hidden Markov model</a> and is one of the most common sequential hierarchical models. Numerous extensions of hidden Markov models have been developed; see the resulting article for more information.

<div class="mw-heading mw-heading2"><h2 id="History">History</h2></div>

Mixture distributions and the problem of mixture decomposition, that is the identification of its constituent components and the parameters thereof, has been cited in the literature as far back as 1846 (Quetelet in McLachlan,<a href="#cite_note-McLachlan_2-18">[18]</a> 2000) although common reference is made to the work of <a href="Karl_Pearson" title="Karl Pearson">Karl Pearson</a> (1894)<a href="#cite_note-Amendola2015-22">[22]</a> as the first author to explicitly address the decomposition problem in characterising non-normal attributes of forehead to body length ratios in female shore crab populations. The motivation for this work was provided by the zoologist <a href="Walter_Frank_Raphael_Weldon" class="mw-redirect" title="Walter Frank Raphael Weldon">Walter Frank Raphael Weldon</a> who had speculated in 1893 (in Tarter and Lock<a href="#cite_note-tart-13">[13]</a>) that asymmetry in the histogram of these ratios could signal evolutionary divergence. Pearson's approach was to fit a univariate mixture of two normals to the data by choosing the five parameters of the mixture such that the empirical moments matched that of the model.

While his work was successful in identifying two potentially distinct sub-populations and in demonstrating the flexibility of mixtures as a moment matching tool, the formulation required the solution of a 9th degree (nonic) polynomial which at the time posed a significant computational challenge.

Subsequent works focused on addressing these problems, but it was not until the advent of the modern computer and the popularisation of <a href="Maximum_Likelihood" class="mw-redirect" title="Maximum Likelihood">Maximum Likelihood</a> (MLE) parameterisation techniques that research really took off.<a href="#cite_note-McLachlan_1-23">[23]</a> Since that time there has been a vast body of research on the subject spanning areas such as <a href="Fishery" title="Fishery">fisheries research</a>, <a href="Agriculture" title="Agriculture">agriculture</a>, <a href="Botany" title="Botany">botany</a>, <a href="Economics" title="Economics">economics</a>, <a href="Medicine" title="Medicine">medicine</a>, <a href="Genetics" title="Genetics">genetics</a>, <a href="Psychology" title="Psychology">psychology</a>, <a href="Palaeontology" class="mw-redirect" title="Palaeontology">palaeontology</a>, <a href="Electrophoresis" title="Electrophoresis">electrophoresis</a>, <a href="Finance" title="Finance">finance</a>, <a href="Geology" title="Geology">geology</a> and <a href="Zoology" title="Zoology">zoology</a>.<a href="#cite_note-titter_1-24">[24]</a>

<div class="mw-heading mw-heading3"><h3 id="Mixture">Mixture</h3></div>

<ul><li><a href="Mixture_density" class="mw-redirect" title="Mixture density">Mixture density</a></li>

<li><a href="Mixture_(probability)" title="Mixture (probability)">Mixture (probability)</a></li>

<li>Flexible Mixture Model (FMM)</li>

<li><a href="Subspace_Gaussian_mixture_model" title="Subspace Gaussian mixture model">Subspace Gaussian mixture model</a></li>

<li><a href="Giry_monad" title="Giry monad">Giry monad</a></li>

<li><a href="HELIOS_Hybrid_Evaluation_of_Lifecycle_and_Impact_of_Outstanding_Science" title="HELIOS Hybrid Evaluation of Lifecycle and Impact of Outstanding Science">HELIOS Hybrid Evaluation of Lifecycle and Impact of Outstanding Science</a></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Hierarchical_models">Hierarchical models</h3></div>

<ul><li><a href="Graphical_model" title="Graphical model">Graphical model</a></li>

<li><a href="Hierarchical_Bayes_model" class="mw-redirect" title="Hierarchical Bayes model">Hierarchical Bayes model</a></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Outlier_detection">Outlier detection</h3></div>

<ul><li><a href="RANSAC" class="mw-redirect" title="RANSAC">RANSAC</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFPalHeumann2024" class="citation journal cs1">Pal, Samyajoy; Heumann, Christian (2024). "Flexible Multivariate Mixture Models: A Comprehensive Approach for Modeling Mixtures of Non‐Identical Distributions". International Statistical Review. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Finsr.12593">10.1111/insr.12593</a>.</cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> <cite id="CITEREFChatzisKosmopoulosVarvarigou2008" class="citation journal cs1">Chatzis, Sotirios P.; Kosmopoulos, Dimitrios I.; Varvarigou, Theodora A. (2008). "Signal Modeling and Classification Using a Robust Latent Space Model Based on t Distributions". IEEE Transactions on Signal Processing. 56 (3): 949–963. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2008ITSP...56..949C">2008ITSP...56..949C</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTSP.2007.907912">10.1109/TSP.2007.907912</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:15583243">15583243</a>.</cite>

</li>

<cite id="CITEREFYu2012" class="citation journal cs1">Yu, Guoshen (2012). "Solving Inverse Problems with Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity". IEEE Transactions on Image Processing. 21 (5): 2481–2499. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1006.3056">1006.3056</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012ITIP...21.2481G">2012ITIP...21.2481G</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2Ftip.2011.2176743">10.1109/tip.2011.2176743</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/22180506">22180506</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:479845">479845</a>.</cite>

</li>

<li id="cite_note-4"><a href="#cite_ref-4">^</a> Dinov, ID. "<a rel="nofollow" class="external text" href="http://repositories.cdlib.org/socr/EM_MM/">Expectation Maximization and Mixture Modeling Tutorial</a>". <a rel="nofollow" class="external text" href="http://repositories.cdlib.org/escholarship">California Digital Library</a>, Statistics Online Computational Resource, Paper EM_MM, <a rel="nofollow" class="external free" href="http://repositories.cdlib.org/socr/EM_MM">http://repositories.cdlib.org/socr/EM_MM</a>, December 9, 2008

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFBishop2006" class="citation book cs1">Bishop, Christopher (2006). Pattern recognition and machine learning. New York: Springer. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-387-31073-2</bdi>.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> Spall, J. C. and Maryak, J. L. (1992). "A feasible Bayesian estimator of quantiles for projectile accuracy from non-i.i.d. data." Journal of the American Statistical Association, vol. 87 (419), pp. 676–681. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2290205">2290205</a>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFAmruthnathGupta2018" class="citation book cs1">Amruthnath, Nagdev; Gupta, Tarun (2018-02-02). <a rel="nofollow" class="external text" href="https://www.researchgate.net/publication/322900854">Fault Class Prediction in Unsupervised Learning using Model-Based Clustering Approach</a>. Unpublished. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.13140%2Frg.2.2.22085.14563">10.13140/rg.2.2.22085.14563</a>.</cite>

</li>

<li id="cite_note-8"><a href="#cite_ref-8">^</a> <cite id="CITEREFAmruthnathGupta2018" class="citation book cs1">Amruthnath, Nagdev; Gupta, Tarun (2018-02-01). <a rel="nofollow" class="external text" href="https://www.researchgate.net/publication/322869981">A Research Study on Unsupervised Machine Learning Algorithms for Fault Detection in Predictive Maintenance</a>. Unpublished. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.13140%2Frg.2.2.28822.24648">10.13140/rg.2.2.28822.24648</a>.</cite>

</li>

<cite id="CITEREFShen2006" class="citation journal cs1">Shen, Jianhong (Jackie) (2006). <a rel="nofollow" class="external text" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2324060">"A stochastic-variational model for soft Mumford-Shah segmentation"</a>. International Journal of Biomedical Imaging. 2006: 2–16. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2006IJBI.200649515H">2006IJBI.200649515H</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1155%2FIJBI%2F2006%2F92329">10.1155/IJBI/2006/92329</a>. <a href="PMC_(identifier)" class="mw-redirect" title="PMC (identifier)">PMC</a> <a rel="nofollow" class="external text" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2324060">2324060</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/23165059">23165059</a>.</cite>

</li>

<cite id="CITEREFMyronenkoSong2010" class="citation journal cs1">Myronenko, Andriy; Song, Xubo (2010). "Point set registration: Coherent point drift". IEEE Trans. Pattern Anal. Mach. Intell. 32 (12): 2262–2275. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/0905.2635">0905.2635</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTPAMI.2010.46">10.1109/TPAMI.2010.46</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/20975122">20975122</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:10809031">10809031</a>.</cite>

</li>

<cite id="CITEREFRavikumarGooyaCimenFrangi2018" class="citation journal cs1">Ravikumar, Nishant; Gooya, Ali; Cimen, Serkan; Frangi, Alexjandro; Taylor, Zeike (2018). <a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.media.2017.11.012">"Group-wise similarity registration of point sets using Student's t-mixture model for statistical shape models"</a>. Med. Image Anal. 44: 156–176. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.media.2017.11.012">10.1016/j.media.2017.11.012</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/29248842">29248842</a>.</cite>

</li>

<cite id="CITEREFBayerRavikumarStrumiaTong2018" class="citation conference cs1">Bayer, Siming; Ravikumar, Nishant; Strumia, Maddalena; Tong, Xiaoguang; Gao, Ying; Ostermeier, Martin; Fahrig, Rebecca; Maier, Andreas (2018). <a rel="nofollow" class="external text" href="https://www.miccai2018.org/en/">"Intraoperative brain shift compensation using a hybrid mixture model"</a>. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. Granada, Spain: Springer, Cham. pp. 116–124. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-030-00937-3_14">10.1007/978-3-030-00937-3_14</a>.</cite>

</li>

<li id="cite_note-tart-13">^ <a href="#cite_ref-tart_13-0">a</a> <a href="#cite_ref-tart_13-1">b</a> <a href="#cite_ref-tart_13-2">c</a> <cite id="CITEREFTarter1993" class="citation cs2">Tarter, Michael E. (1993), Model Free Curve Estimation, Chapman and Hall</cite>

</li>

<li id="cite_note-Jain-14">^ <a href="#cite_ref-Jain_14-0">a</a> <a href="#cite_ref-Jain_14-1">b</a> <cite id="CITEREFFigueiredoJain2002" class="citation journal cs1">Figueiredo, M.A.T.; Jain, A.K. (March 2002). "Unsupervised Learning of Finite Mixture Models". IEEE Transactions on Pattern Analysis and Machine Intelligence. 24 (3): 381–396. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.362.9811">10.1.1.362.9811</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2F34.990138">10.1109/34.990138</a>.</cite>

</li>

<cite id="CITEREFMcWilliamLoh2008" class="citation cs2">McWilliam, N.; Loh, K. (2008), Incorporating Multidimensional Tail-Dependencies in the Valuation of Credit Derivatives (Working Paper)</cite> <a rel="nofollow" class="external autonumber" href="http://www.misys.com/cds-portlets/digitalAssets/4/2797_CDsAndTailDep_forPublication_final1.pdf">[1]</a>

</li>

<li id="cite_note-dempster1977-16">^ <a href="#cite_ref-dempster1977_16-0">a</a> <a href="#cite_ref-dempster1977_16-1">b</a> <cite id="CITEREFDempsterLairdRubin1977" class="citation journal cs1">Dempster, A.P.; Laird, N.M.; Rubin, D.B. (1977). "Maximum Likelihood from Incomplete Data via the EM Algorithm". Journal of the Royal Statistical Society, Series B. 39 (1): 1–38. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.163.7580">10.1.1.163.7580</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fj.2517-6161.1977.tb01600.x">10.1111/j.2517-6161.1977.tb01600.x</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2984875">2984875</a>.</cite>

</li>

<li id="cite_note-XuJordam-17"><a href="#cite_ref-XuJordam_17-0">^</a> <cite id="CITEREFXuJordan1996" class="citation journal cs1">Xu, L.; Jordan, M.I. (January 1996). "On Convergence Properties of the EM Algorithm for Gaussian Mixtures". Neural Computation. 8 (1): 129–151. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1162%2Fneco.1996.8.1.129">10.1162/neco.1996.8.1.129</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<a rel="nofollow" class="external text" href="https://hdl.handle.net/10338.dmlcz%2F135225">10338.dmlcz/135225</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:207714252">207714252</a>.</cite>

</li>

<li id="cite_note-McLachlan_2-18">^ <a href="#cite_ref-McLachlan_2_18-0">a</a> <a href="#cite_ref-McLachlan_2_18-1">b</a> <cite id="CITEREFMcLachlan2000" class="citation cs2">McLachlan, G.J. (2000), Finite Mixture Models, Wiley</cite>

</li>

<li id="cite_note-botev2004global-19"><a href="#cite_ref-botev2004global_19-0">^</a> <cite id="CITEREFBotev,_Z.I.Kroese,_D.P.2004" class="citation book cs1">Botev, Z.I.; <a href="Dirk_Kroese" title="Dirk Kroese">Kroese, D.P.</a> (2004). "Global Likelihood Optimization Via the Cross-Entropy Method, with an Application to Mixture Models". Proceedings of the 2004 Winter Simulation Conference, 2004. Vol. 1. pp. 517–523. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.331.2319">10.1.1.331.2319</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FWSC.2004.1371358">10.1109/WSC.2004.1371358</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-7803-8786-7</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:6880171">6880171</a>.</cite>

</li>

<li id="cite_note-day-20"><a href="#cite_ref-day_20-0">^</a> <cite id="CITEREFDay1969" class="citation journal cs1">Day, N. E. (1969). "Estimating the Components of a Mixture of Normal Distributions". Biometrika. 56 (3): 463–474. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F2334652">10.2307/2334652</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2334652">2334652</a>.</cite>

</li>

<li id="cite_note-wang-21"><a href="#cite_ref-wang_21-0">^</a> <cite id="CITEREFWang2001" class="citation cs2">Wang, J. (2001), "Generating daily changes in market variables using a multivariate mixture of normal distributions", Proceedings of the 33rd Winter Conference on Simulation: 283–289</cite>

</li>

<li id="cite_note-Amendola2015-22"><a href="#cite_ref-Amendola2015_22-0">^</a> <cite id="CITEREFAméndola2015" class="citation journal cs1">Améndola, Carlos; et al. (2015). "Moment varieties of Gaussian mixtures". Journal of Algebraic Statistics. 7. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1510.04654">1510.04654</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2015arXiv151004654A">2015arXiv151004654A</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.18409%2Fjas.v7i1.42">10.18409/jas.v7i1.42</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:88515304">88515304</a>.</cite>

</li>

<li id="cite_note-McLachlan_1-23"><a href="#cite_ref-McLachlan_1_23-0">^</a> <cite id="CITEREFMcLachlanBasford1988" class="citation cs2">McLachlan, G.J.; Basford, K.E. (1988), "Mixture Models: inference and applications to clustering", Statistics: Textbooks and Monographs, <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1988mmia.book.....M">1988mmia.book.....M</a></cite>

</li>

<li id="cite_note-titter_1-24"><a href="#cite_ref-titter_1_24-0">^</a> <a href="#CITEREFTitteringtonSmithMakov1985">Titterington, Smith & Makov 1985</a>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Books_on_mixture_models">Books on mixture models</h3></div>

<ul><li><cite id="CITEREFEverittHand1981" class="citation book cs1">Everitt, B.S.; Hand, D.J. (1981). Finite mixture distributions. Chapman & Hall. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-412-22420-1</bdi>.</cite></li>

<li><cite id="CITEREFLindsay1995" class="citation book cs1"><a href="Bruce_G._Lindsay" title="Bruce G. Lindsay">Lindsay, B. G.</a> (1995). Mixture Models: Theory, Geometry, and Applications. NSF-CBMS Regional Conference Series in Probability and Statistics. Vol. 5. Hayward: Institute of Mathematical Statistics.</cite></li>

<li><cite id="CITEREFMarinMengersenRobert2011" class="citation book cs1">Marin, J.M.; <a href="Kerrie_Mengersen" title="Kerrie Mengersen">Mengersen, K.</a>; <a href="Christian_Robert" title="Christian Robert">Robert, C. P.</a> (2011). <a rel="nofollow" class="external text" href="http://www.ceremade.dauphine.fr/%7Exian/mixo.pdf">"Bayesian modelling and inference on mixtures of distributions"</a> (PDF). In Dey, D.; Rao, C.R. (eds.). Essential Bayesian models. Handbook of statistics: Bayesian thinking - modeling and computation. Vol. 25. Elsevier. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>9780444537324</bdi>.</cite></li>

<li><cite id="CITEREFMcLachlanPeel2000" class="citation book cs1">McLachlan, G.J.; Peel, D. (2000). <a rel="nofollow" class="external text" href="https://archive.org/details/finitemixturemod00geof">Finite Mixture Models</a>. Wiley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-00626-8</bdi>.</cite></li>

<li><cite id="CITEREFPressTeukolskyVetterlingFlannery2007" class="citation book cs1">Press, WH; Teukolsky, SA; Vetterling, WT; Flannery, BP (2007). <a rel="nofollow" class="external text" href="http://numerical.recipes/book">"Section 16.1. Gaussian Mixture Models and k-Means Clustering"</a>. Numerical Recipes: The Art of Scientific Computing (3rd ed.). New York: Cambridge University Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-521-88068-8</bdi>.</cite></li>

<li><cite id="CITEREFTitteringtonSmithMakov1985" class="citation book cs1">Titterington, D.; Smith, A.; Makov, U. (1985). Statistical Analysis of Finite Mixture Distributions. Wiley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-90763-3</bdi>.</cite></li>

<li><cite id="CITEREFYaoXiang2024" class="citation book cs1">Yao, W.; Xiang, S. (2024). <a rel="nofollow" class="external text" href="https://www.routledge.com/Mixture-Models-Parametric-Semiparametric-and-New-Directions/Yao-Xiang/p/book/9780367481827">Mixture Models: Parametric, Semiparametric, and New Directions</a>. Chapman & Hall/CRC Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0367481827</bdi>.</cite></li></ul>

<div class="mw-heading mw-heading3"><h3 id="Application_of_Gaussian_mixture_models">Application of Gaussian mixture models</h3></div>

<ol><li><cite id="CITEREFReynoldsRose1995" class="citation journal cs1">Reynolds, D.A.; Rose, R.C. (January 1995). "Robust text-independent speaker identification using Gaussian mixture speaker models". IEEE Transactions on Speech and Audio Processing. 3 (1): 72–83. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2F89.365379">10.1109/89.365379</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:7319345">7319345</a>.</cite></li>

<li><cite id="CITEREFPermuterFrancosJermyn2003" class="citation conference cs1">Permuter, H.; Francos, J.; Jermyn, I.H. (2003). Gaussian mixture models of texture and colour for image database retrieval. IEEE <a href="International_Conference_on_Acoustics%2C_Speech%2C_and_Signal_Processing" title="International Conference on Acoustics, Speech, and Signal Processing">International Conference on Acoustics, Speech, and Signal Processing</a>, 2003. Proceedings (ICASSP '03). <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FICASSP.2003.1199538">10.1109/ICASSP.2003.1199538</a>.</cite>

<ul><li><cite id="CITEREFPermuterFrancosJermyn2006" class="citation journal cs1">Permuter, Haim; Francos, Joseph; Jermyn, Ian (2006). <a rel="nofollow" class="external text" href="http://dro.dur.ac.uk/16022/1/16022.pdf">"A study of Gaussian mixture models of color and texture features for image classification and segmentation"</a> (PDF). Pattern Recognition. 39 (4): 695–706. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2006PatRe..39..695P">2006PatRe..39..695P</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.patcog.2005.10.028">10.1016/j.patcog.2005.10.028</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:8530776">8530776</a>.</cite></li></ul></li>

<li><cite id="CITEREFLemke2005" class="citation book cs1">Lemke, Wolfgang (2005). Term Structure Modeling and Estimation in a State Space Framework. Springer Verlag. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-540-28342-3</bdi>.</cite></li>

<li><cite id="CITEREFBrigoMercurio2001" class="citation conference cs1"><a href="Damiano_Brigo" title="Damiano Brigo">Brigo, Damiano</a>; <a href="Fabio_Mercurio" title="Fabio Mercurio">Mercurio, Fabio</a> (2001). Displaced and Mixture Diffusions for Analytically-Tractable Smile Models. Mathematical Finance – Bachelier Congress 2000. Proceedings. Springer Verlag.</cite></li>

<li><cite id="CITEREFBrigoMercurio2002" class="citation journal cs1">Brigo, Damiano; Mercurio, Fabio (June 2002). "Lognormal-mixture dynamics and calibration to market volatility smiles". International Journal of Theoretical and Applied Finance. 5 (4): 427. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.210.4165">10.1.1.210.4165</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1142%2FS0219024902001511">10.1142/S0219024902001511</a>.</cite></li>

<li><cite id="CITEREFSpallMaryak1992" class="citation journal cs1">Spall, J. C.; Maryak, J. L. (1992). "A feasible Bayesian estimator of quantiles for projectile accuracy from non-i.i.d. data". Journal of the American Statistical Association. 87 (419): 676–681. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1080%2F01621459.1992.10475269">10.1080/01621459.1992.10475269</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2290205">2290205</a>.</cite></li>

<li><cite id="CITEREFAlexander2004" class="citation journal cs1">Alexander, Carol (December 2004). <a rel="nofollow" class="external text" href="http://www.carolalexander.org/publish/download/JournalArticles/PDFs/JBF2004.pdf">"Normal mixture diffusion with uncertain volatility: Modelling short- and long-term smile effects"</a> (PDF). Journal of Banking & Finance. 28 (12): 2957–80. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.jbankfin.2003.10.017">10.1016/j.jbankfin.2003.10.017</a>.</cite></li>

<li><cite id="CITEREFStylianouPantazisCaldereroLarroy2005" class="citation conference cs1">Stylianou, Yannis; Pantazis, Yannis; Calderero, Felipe; Larroy, Pedro; Severin, Francois; Schimke, Sascha; Bonal, Rolando; Matta, Federico; Valsamakis, Athanasios (2005). <a rel="nofollow" class="external text" href="http://www.enterface.net/enterface05/docs/results/reports/project5.pdf">GMM-Based Multimodal Biometric Verification</a> (PDF).</cite></li>

<li><cite id="CITEREFChenAdebomiOlusayoKulesza2010" class="citation conference cs1">Chen, J.; Adebomi, 0.E.; Olusayo, O.S.; Kulesza, W. (2010). The Evaluation of the Gaussian Mixture Probability Hypothesis Density approach for multi-target tracking. IEEE International Conference on Imaging Systems and Techniques, 2010. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FIST.2010.5548541">10.1109/IST.2010.5548541</a>.</cite><code class="cs1-code">{{cite conference}}</code>: CS1 maint: numeric names: authors list (link)</li></ol>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li><cite id="CITEREFNielsen2012" class="citation book cs1">Nielsen, Frank (23 March 2012). "K-MLE: A fast algorithm for learning statistical mixture models". 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 869–872. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1203.5181">1203.5181</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012arXiv1203.5181N">2012arXiv1203.5181N</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FICASSP.2012.6288022">10.1109/ICASSP.2012.6288022</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-4673-0046-9</bdi>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:935615">935615</a>.</cite></li>

<li>The <a rel="nofollow" class="external text" href="http://wiki.stat.ucla.edu/socr/index.php/SOCR_EduMaterials_Activities_2D_PointSegmentation_EM_Mixture">SOCR demonstrations of EM and Mixture Modeling</a></li>

<li><a rel="nofollow" class="external text" href="http://www.csse.monash.edu.au/~dld/mixturemodel.html">Mixture modelling page</a> (and the <a rel="nofollow" class="external text" href="http://www.csse.monash.edu.au/~dld/Snob.html">Snob</a> program for <a href="Minimum_Message_Length" class="mw-redirect" title="Minimum Message Length">Minimum Message Length</a> (<a href="Minimum_Message_Length" class="mw-redirect" title="Minimum Message Length">MML</a>) applied to finite mixture models), maintained by D.L. Dowe.</li>

<li><a rel="nofollow" class="external text" href="http://www.pymix.org">PyMix</a> – Python Mixture Package, algorithms and data structures for a broad variety of mixture model based data mining applications in Python</li>

<li><a rel="nofollow" class="external text" href="http://scikit-learn.org/stable/modules/mixture.html">sklearn.mixture</a> – A module from the <a href="Scikit-learn" title="Scikit-learn">scikit-learn</a> Python library for learning Gaussian Mixture Models (and sampling from them), previously packaged with <a href="SciPy" title="SciPy">SciPy</a> and now packaged as a <a rel="nofollow" class="external text" href="https://scikits.appspot.com/">SciKit</a></li>

<li><a rel="nofollow" class="external text" href="http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=18785&objectType=FILE">GMM.m</a> Matlab code for GMM Implementation</li>

<li><a rel="nofollow" class="external text" href="http://stat.duke.edu/gpustatsci/software.html">GPUmix</a> C++ implementation of Bayesian Mixture Models using EM and MCMC with 100x speed acceleration using GPGPU.</li>

<li><a rel="nofollow" class="external autonumber" href="https://www.cs.ru.nl/~ali/index_files/EM.m">[2]</a> Matlab code for GMM Implementation using EM algorithm</li>

<li><a rel="nofollow" class="external autonumber" href="https://vincentfpgarcia.github.com/jMEF/">[3]</a> jMEF: A Java open source library for learning and processing mixtures of exponential families (using duality with Bregman divergences). Includes a Matlab wrapper.</li>

<li>Very Fast and clean C implementation of the <a rel="nofollow" class="external text" href="https://github.com/juandavm/em4gmm">Expectation Maximization</a> (EM) algorithm for estimating <a rel="nofollow" class="external text" href="https://github.com/juandavm/em4gmm">Gaussian Mixture Models</a> (GMMs).</li>

<li><a rel="nofollow" class="external text" href="https://cran.r-project.org/web/packages/mclust/index.html">mclust</a> is an R package for mixture modeling.</li>

<li><a rel="nofollow" class="external text" href="https://github.com/thaines/helit/tree/master/dpgmm">dpgmm</a> Pure Python Dirichlet process Gaussian mixture model implementation (variational).</li>

<li><a rel="nofollow" class="external text" href="https://mpatacchiola.github.io/blog/2020/07/31/gaussian-mixture-models.html">Gaussian Mixture Models</a> Blog post on Gaussian Mixture Models trained via Expectation Maximization, with an implementation in Python.</li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-08-07" href="https://en.wikipedia.org/wiki/?title=Mixture_model&oldid=1304657826">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>